Apparatus and method for inputting reflected light image of a target object

ABSTRACT

An area image sensor outputs the difference between charges received by light-receiving cells arranged in an array pattern. A system controller generates a timing signal for generating a pulse or modulation signal. A control signal generator generates a control signal for separately controlling the light-receiving timings of the light-receiving cells of the area image sensor on the basis of the timing signal from the system controller. A light emitting controller controls a light source to generate light, the intensity of which changes on the basis of the timing signal from the system controller. A reflected light image processor extracts a reflected image of an object from the difference outputted from the area image sensor.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional application of Ser. No. 09/268,645filed Mar. 16, 1999, and claims priority to the Japanese Application No.10-066382, filed on Mar. 17, 1998, the entire contents of which beingincorporated herein by reference.

BACKGROUND OF THE INVENTION

This application is based on Japanese Patent Application No. 10-66382,filed Mar. 17, 1998, the contents of which are incorporated herein byreference.

The present invention relates to an information input apparatus andmethod for inputting information in a three-dimensional space, and to arecording medium.

As an input device to a computer, a mouse is prevalently used. However,the mouse is used to merely attain roles of a two-dimensional pointingdevice such as movement of the cursor, selection of a menu, and thelike. Since information the mouse can process in two-dimensionalinformation, the mouse can hardly select, e.g., an object with a depthin a three-dimensional space. On the other hand, when the mouse is usedto animate a character upon creating an animation, it cannot easilynaturally animate the character. In order to compensate for suchdifficulties in pointing in a three-dimensional space, three-dimensionalpointing devices have been developed. For example, a three-dimensionalpointing device 150 shown in FIG. 1 allows six ways of operations, i.e.,pushing a central round portion forward, pressing the center of thatportion, pressing the rear end of that portion, lifting the entireportion upward, turning the entire portion clockwise, and turning theentire portion counterclockwise, and has six degrees of freedom. Byassigning these six degrees of freedom to various instructions, theposition (x, y, z) and directions (x-, y-, and z-axes) of a cursor inthree-dimensional space can be controlled, or the view point position(x, y, z) and directions (x-, y-, and z-axes) with respect to thethree-dimensional space can be controlled.

However, when this device is operated actually, the cursor or view pointcannot be desirably controlled. For example, when the operator wants toturn the round portion clockwise or counterclockwise, he or she maypress its forward or rear end, and the cursor or view point may move inan unexpected direction.

In place of such three-dimensional pointing device, devices that caninput instructions using hand or body actions have been developed. Suchdevices are called, e.g., a data glove, data suit, cyber glove, and thelike. For example, the data glove is a glove-like device, and opticalfibers run on its surface. Each optical fiber runs to a joint of eachfinger, and upon bending the finger, the transmission state of lightchanges. By measuring the transmission state of light, the bent level ofthe joint of each finger can be detected. The position of the handitself in the three-dimensional space is measured by a magnetic sensorattached to the back of the hand. If an action is assigned to a giveninstruction (e.g., if the index finger is pointed up, a forward movementinstruction is issued), the operator can walk in the three-dimensionalspace by variously changing the view point using the data glove(walkthrough).

However, some problems must be solved. Such device is expensive, and canhardly be used for home use. Since the angle of the finger joint ismeasured, even when, for example, stretching only the index finger andbending other fingers is defined as a forward movement instruction,stretching a finger includes various states. That is, since the secondjoint of the index finger rarely makes 180°, it is different torecognize the stretched state except for such 180° state of the indexfinger, unless a given margin is assured. Since the operator must wearthe data glove, his or her natural movement is disturbed. Every time theoperator wears the data glove, he or she must calibrate the transmissionstate of light in correspondence with the stretched and bent fingerstates, resulting in troublesome operations. Since optical fibers areused, failures such as disconnection of fibers may take place aftercontinuous use of the data glove, and the data glove has a durability aslow as an expendable. Despite the fact the data glove is such expensive,troublesome device, if the glove size does not just fit with theoperator's hand, the input value may deviate from the calibrated valueduring use due to slippage of the glove, and delicate hand actions canhardly be recognized. Owing to various problems described above, thedata glove has not so prevailed contrary to initial expectation althoughit served as a trigger device of the VR (virtual reality) technology.For this reason, the data glove is still expensive, and has manyproblems in terms of its use.

By contrast, some studies have been made to input hand and body actionswithout wearing any special devices such as a data glove. For example, amethod of recognizing hand shape by analyzing a moving image such as avideo image has been studied.

However, with such method, it is very hard to extract an objective imageportion (e.g., in case of hand action recognition, a hand image alone)from the background image. For example, assume that an objective imageis extracted using colors. Since the hand has skin color, only a skincolor portion may be extracted. However, if a beige clothing article orwall is present as a background, it is hard to recognize skin color.Even when beige is distinguished from skin color by adjustment, ifillumination changes, the color tone also changes. Hence, it isdifficult to steadily extract a skin color portion.

In order to avoid such problems, a method that facilitates extraction byimposing a constraint on the background image, e.g., by placing a bluemat on the background may be used. Alternatively, a method that colorsfinger tips to easily extract them from the background or makes theoperator wear color rings may be used. However, such constraints are notpractical; they are used for experimental purposes but are not put intopractical applications.

The above-mentioned video image recognition such as extraction and thelike requires a very large computation amount. For this reason, existingpersonal computers cannot process all video images (as large as 30images per sec) in real time. Hence, it is hard to attain motion captureby video image processing in real time.

A device called a range finder for inputting a distant image is known.The typical principle of the range finder is to irradiate an object withspot light or slit light and obtain a distant image based on theposition where the light reflected by the object is received by theprinciple of triangulation. The range finder mechanically scans spotlight or slit light to obtain two-dimensional distance information. Thisdevice can generate a distant image with very high precision, butrequires a large-scale arrangement, resulting in high cost. Also, a longinput time is required, and it is difficult for this device to processinformation in real time.

A device for detecting a color marker or light-emitting unit attached toa hand or body portion from an image, and capturing the shape, motion,and the like of the hand or body portion may be used, and has alreadybeen put into some applications. However, the device has a seriousdemerit of user's inconvenience, since the user must wear the deviceupon every operation, and the application range is limited very much. Asin the example of the data glove, when the user wears the device on hisor her movable portion such as a hand, the durability problem is oftenposed.

The problems in a conventional camera technique will be explained belowin addition to the aforementioned input devices. With the conventionalcamera technique, in order to synthesize (chromakey) a character with abackground, a character image must be photographed in front of a blueback to facilitate character extraction. For this reason, thephotographing place is limited to, e.g., a studio that can photograph animage in front of a blue back. Alternatively, in order to extract acharacter from an image photographed in a non-blue back state, thecharacter extraction range must be manually edited in units of frames,resulting in very cumbersome operations.

Similarly, in order to generate a character in a three-dimensionalspace, a three-dimensional model is created in advance, and a photographof the character is pasted to the model (texture mapping). However,creation of a three-dimensional model and texture mapping are tediousoperations and are rarely used other than applications such as movieproduction that justifies extravagant cost needed.

In order to solve these problems, for example, a technique disclosed inU.S. Ser. No. 08/953,667 (now U.S. Pat. No. 6,144,366) is known. Thistechnique acquires a distant image by extracting a reflected lightimage. However, this technique cannot obtain hue information of anobject since it extracts the reflected light image. For this reason, twodifferent types of cameras, i.e., a conventional imaging camera and acamera for extracting a reflected light image, are required.

BRIEF SUMMARY OF THE INVENTION

It is an object of the present invention to provide an information inputapparatus and method that can acquire a reflected light image using aversatile image sensor, and a recording medium.

It is another object of the present invention to provide an informationinput apparatus and method which can attain high-level image processingsuch as extraction of an object image alone from the background in anormal image, and the like, and a recording medium.

In order to achieve the above objects, according to the first aspect ofthe present invention, an information input apparatus comprises: a lightemitter for irradiating an object with light; an area image sensor foroutputting a difference between charges received by light-receivingcells arranged in an array pattern from a reflected light of the objectcaused by the light emitter irradiating the object with light; a timingsignal generator for generating a timing signal comprised of a pulsesignal or a modulation signal for controlling an intensity of light ofthe light emitter; a control signal generator for generating a controlsignal for individually controlling light-receiving timings of thelight-receiving cells of the area image sensor on the basis of thetiming signal from the timing signal generator; and image processingsection for extracting a reflected light image of the object from thedifference outputted from the area image sensor.

According to the second aspect of the present invention, an informationinput apparatus comprises: a timing signal generator for generating atiming signal comprised of a pulse signal or a modulation signal; alight emitter for emitting light, an intensity of which changes on thebasis of the timing signal from the timing signal generator; firstlight-receiving section for receiving light emitted by the light emitterand reflected by an object in synchronism with the timing signal fromthe timing signal generator; and second light-receiving section forreceiving light other than the light emitted by the light emitter andreflected by the object.

According to the third aspect of the present invention, an informationinput method comprises the steps of: generating a pulse signal or amodulation signal; generating, on the basis of the pulse or modulationsignal, a control signal for separately controlling light-receivingtimings of light-receiving cells of an area image sensor for obtaining adifference between charges received by light-receiving cells which arearranged in an array pattern; emitting light, an intensity of whichchanges on the basis of the generated control signal; and detecting alight image reflected by an object of the emitted light.

According to the fourth aspect of the present invention, an informationinput method comprises the steps of: generating a pulse signal ormodulation signal; emitting light, an intensity of which changes on thebasis of the pulse or modulation signal; and receiving light reflectedby an object of the emitted light and light other than the reflectedlight in synchronism with the pulse or modulation signal.

According to the fifth aspect of the present invention, an article ofmanufacture comprises: a computer usable medium having computer readableprogram code means embodied therein for causing an area image sensor forobtaining a difference between charges received by light-receiving cellswhich are arranged in an array pattern to be controlled, the computerreadable program code means in the article of manufacture comprising:computer readable program code means for causing a computer to generatea pulse signal or a modulation signal; computer readable program codemeans for causing a computer to generate a control signal for separatelycontrolling light-receiving timings of the light-receiving cells of thearea image sensor on the basis of the pulse or modulation signal;computer readable program code means for causing a computer to cause alight emitter to emit light, an intensity of which changes on the basisof the generated pulse signal or modulation signal; and computerreadable program code means for causing a computer to extract a lightimage reflected by an object of the emitted light from the differenceoutputted from the area image sensor.

According to the present invention, since a reflected light image can beacquired using a versatile image sensor, i.e., the versatile imagesensor can be used, a cost reduction of the apparatus can be realized.

According to the present invention, high-level image processing such asextraction of an object image alone from the background in a normalimage, and the like can be easily implemented.

Furthermore, according to the present invention, a reflected image andan image based on other light components can be simultaneously obtained.

Additional objects and advantages of the invention will be set forth inthe description which follows, and in part will be obvious from thedescription, or may be learned by practice of the invention. The objectsand advantages of the invention may be realized and obtained by means ofthe instrumentalities and combinations particularly pointed outhereinafter.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate presently preferred embodiments ofthe invention, and together with the general description given above andthe detailed description of the preferred embodiments given below, serveto explain the principles of the invention.

FIG. 1 is a perspective view showing an example of a conventionalthree-dimensional data input device;

FIG. 2 is a block diagram showing an example of the arrangement foracquiring a reflected light image using an image sensor;

FIG. 3 is a block diagram showing an example of the arrangement foracquiring a reflected light image using an image sensor in more detail;

FIG. 4 is a flow chart showing the operation for acquiring a reflectedlight image using an image sensor;

FIG. 5 shows an example of the acquired reflected light image;

FIG. 6 is a block diagram showing an example of the arrangement of anapparatus that can simultaneously acquire an image based on objectreflected light and an image based on other light components;

FIG. 7 is a block diagram showing an example of the arrangement of anapparatus that can simultaneously acquire an image based on objectreflected light and an image based on other light components;

FIG. 8 is a block diagram showing an example of the arrangement of anapparatus that can simultaneously acquire an image based on objectreflected light and an image based on other light components;

FIG. 9 shows an example of the filter arrangement on an image sensor ofthe apparatus that can simultaneously acquire an image based on objectreflected light and an image based on other light components;

FIG. 10 shows an example of the filter arrangement on an image sensor ofthe apparatus that can simultaneously acquire an image based on objectreflected light and an image based on other light components;

FIG. 11 is a block diagram showing an example of the arrangement of anapparatus that can simultaneously acquire an image based on objectreflected light and an image based on other light components;

FIG. 12 is a block diagram showing an example of the arrangement of anapparatus that can simultaneously acquire an image based on objectreflected light and an image based on other light components;

FIG. 13 shows an example of the filter arrangement on an image sensor ofthe apparatus that can simultaneously acquire an image based on objectreflected light and an image based on other light components; and

FIG. 14 shows an example of the filter arrangement on an image sensor ofthe apparatus that can simultaneously acquire an image based on objectreflected light and an image based on other light components.

DETAILED DESCRIPTION OF THE INVENTION

The preferred embodiments of the present invention will be explainedhereinafter with the aid of the accompanying drawings.

FIG. 2 shows an example of the arrangement of an apparatus for acquiringa reflected light image using an image sensor. A light emittingcontroller 8 controls a light source 7 to emit light in a predeterminedlight emitting pattern on the basis of a timing signal comprised of apulse signal or modulation signal generated by a system controller 11.More specifically, the light source emits one-shot pulse light perframe. The wavelength of light is not particularly limited, but a lightsource that emits light such as near infrared light which is containedin external light (illumination light, sunlight), has low power, anddoes not glare, is preferably used.

The light source 7 preferably comprises, e.g., an LED which stablyoperates for a long period of time, and responds at high speed. Alight-receiving sensor 1 operates to sense at least twice in synchronismwith the light emitted by the light source 7. The sensor 1 operatessimultaneously with light emission to receive light containing objectreflected light 5 of light 14 emitted by the light source. The sensor 1also operates when the light source ceases to emit any light, so as toreceive light that does not contain any object reflected light of thelight emitted by the light source. This sensor can output the differencebetween these two imaging results. That is, by obtaining thisdifference, the object reflected light alone of the light emitted by thelight source is output. An intensity of the light emitted by the lightsource is controlled based on the timing signal.

A control signal generator 10 generates control signals for controllinglight reception of individual light-receiving cells of thelight-receiving sensor 1 on the basis of the pulse or modulation signalgenerated by the system controller 11, and supplies these signals to thelight-receiving sensor 1.

A filter 2 that intercepts unwanted light is inserted between a lens 3and the sensor 1. For example, an optical bandpass filter that passesonly the light source wavelength is inserted. For this reason, most ofunwanted light components 4 such as illumination light do not becomeincident on the sensor. However, since external light such asillumination light or the like normally has light components having thesame wavelength as the light source wavelength, the difference isoutput.

The sensor output is converted into digital data by an A/D converter 12via an analog signal processor 9. A reflected light image processor 13performs various kinds of processing using the digital image data. Uponinterpreting the operator's hand action, the processor 13 extracts atwo- or three-dimensional hand shape, and estimates an action of thehand 6. A detailed description of the processing in this reflected lightimage processor 13 will be omitted in this embodiment.

FIG. 3 shows an image sensor in the light-receiving sensor unit shown inFIG. 2 in more detail. The image sensor shown in FIG. 2 can designatepixel position by a vertical selector 15 and horizontal selector 17 andcan extract an output stored in the designated cell.

A difference circuit 18 can temporarily store a charge amount and canoutput the difference between the stored charge amount and the nextinput charge amount. In this image sensor, imaging operations of evenand odd lines can be separately controlled. The even lines are used ascells that receive light when the light source emits light, and the oddlines are used as cells that receive light when the light source ceasesto emit light. More specifically, the cells in the even lines sense whenthe light source emits light, and the cells in the odd lines sense whenthe light source ceases to emit light. The former cells will be referredto as emission-timing storage cells 20, and the latter cells will bereferred to as non-emission-timing storage cells 19 hereinafter.

The control signal generator 16 generates control signals forcontrolling the vertical selector 15, the difference circuit 18 and thehorizontal section 17. A controller 21 generates control signals tocontrol the signal generator 16, the A/D converter 12 and the reflectedlight image processor 13.

For example, the difference circuit 18 operates as follows. Thedifference circuit 18 reads out contents of non-emission-timing storagecells in the first line, and stores them for one line. Then, thedifference circuit 18 reads out the contents of non-emission-timingstorage cells in the second line, and outputs the difference between thecurrently readout amount and the stored received-light amount for thefirst line. The subsequent processing is the same as that in FIG. 2.

FIG. 4 is a flow chart showing the processing in the arrangement shownin FIG. 3. First, initialization is made (S100). When the LED emitslight (step S101), the even lines receive light (step S102); when theLED ceases to emit light (step S103), the odd lines receive light (stepS104). In this way, as shown in FIG. 3, charges received while light isemitted are stored in the even lines, and those received while light isnot emitted are stored in the odd lines. These charges are read out viathe difference circuit 18.

Initially, the received-light level of one odd line, i.e., at anon-emission timing is stored in the difference circuit (step S106).Then, the received-light level of one even line, i.e., at an emissiontiming are read out to the difference circuit 18 (step S107), and onlythe difference between the levels at the emission and non-emissiontimings is output (step S108). The difference is input to the analogsignal processor 9 (step S109).

Such operation continues until the contents of all the lines aretransferred in the vertical direction S110. Upon completion of transfer,operation for emitting and receiving light is repeated.

With the above-mentioned operation, for example, a reflected light image(a three-dimensional image in practice since the intensity of reflectedlight is inversely proportional to the square of distance) shown in FIG.5 is acquired.

In FIG. 3, the apparatus for acquiring an object reflected light imageof a light source using an image sensor has been described. However,there are requirements for an apparatus that can simultaneously obtain areflected light image from which the object shape can be easilyacquired, and an image taken by a normal camera (to be referred to as avisible light image hereinafter). If such apparatus is realized, a humanbody or face alone can be easily extracted from the visible light image.

FIG. 6 shows an example of the arrangement of an apparatus which cansimultaneously obtain a reflected light image and visible light image.In this embodiment, two different types of image sensors are prepared incorrespondence with the reflected light image and visible light image.

A light-receiving sensor 1 for the reflected light image has thearrangement described above with the aid of FIG. 3. An optical bandpassfilter 2 that passes only the wavelength light of a light source isplaced in front of the image sensor 1. This filter passes only objectreflected light 5 of the light source, and reflects other lightcomponents such as illumination light and its object reflected light 4.

On the other hand, a sensor 22 for the visible light image can use a CCDimage sensor, CMOS image sensor, and the like, which are used in anormal imaging camera. An optical filter 23 that reflects objectreflected light from the light source is normally placed in front of thesensor for the visible light image. However, when the object reflectedlight of the light source does not influence largely, e.g., when aninfrared light cut filter used in an imaging camera suffices or when thesensor for the visible light image has low sensitivity to the lightsource due to its spectral sensitivity characteristics, this opticalfilter may be omitted.

When these two sensors simultaneously operate, higher-level informationsuch as an extracted object can be obtained. Also, this apparatus can beused for simply obtaining a reflected light image or visible light imagealone.

In this arrangement, since the sensor for the reflected light image isplaced in the vicinity of the sensor for the visible light image, imagesnearly free from position errors can be obtained. However, strictlyspeaking, slight position errors are present, and they becomeconspicuous if the object is close to the apparatus.

A control signal generator 24 generates a control signal for controllingthe sensor 22 for the visible light image. An analog signal processor 25processes a reflective image signal and color image signal. A systemcontroller 27 controls the whole system.

FIG. 7 shows an example of the arrangement that can prevent suchposition errors. In this arrangement, an optical system is shared by thereflected light image and visible light image. Light that has passedthrough the optical system is split into object reflected light 29 ofthe light source and other light components 30 by, e.g., a dichroicmirror 28. The dichroic mirror passes light in a specific wavelengthrange, and reflects other light components. The arrangement shown inFIG. 7 uses a mirror that passes the wavelength light of the objectreflected light, and reflects other light components. With thisarrangement, two images having optical axes which accurately match,i.e., a reflected light image and visible light image, can be obtained.

In this example, a dichroic mirror is used. However, the presentinvention is not limited to such specific device but can use any otherdevices which can split light into the object reflected light of thelight source and other light components, and can input them to two imagesensors placed at positions where they do not overlap each other.

In FIGS. 6 and 7, the two sensors respectively for the reflected lightimage and visible light image are used. If such processing can be doneusing a single sensor, further cost reduction can be attained. FIG. 8shows one example of such arrangement. A slidable optical filter 31 isinserted between a lens 32 and sensor 33.

This filter has a bandpass filter that passes only the wavelength lightof the light source on its half portion, and has an optical filter thatintercepts this wavelength light and passes visible light on the otherhalf portion. By sliding this filter, light that reaches the sensor canbe limited to only the object reflected light of the light source orvisible light. In this case, a single image sensor must sense both areflected light image-and visible light image. Such sensor will bedescribed below.

FIG. 9 shows a color filter matrix normally used in a CMOS image sensor.As shown in FIG. 9, this matrix includes “R” cells that pass redcomponents, “G” cells that pass green components, and “B” cells thatpass blue components. When a visible light image is to be sensed, theoutputs from these cells are directly output. During this period, thedifference circuit ceases to operate or is used only for removing fixedpattern noise. The fixed pattern noise is the one that has fixed valuesin units of cells.

In order to obtain a reflected light image, the sensor operation isswitched. For example, only “G” cells are used. In this case, a G1 cellstores a non-emission-timing charge, a G2 cell stores an emission-timingcharge, and their difference can be output using the difference circuit.For this purpose, imaging control of G cells must be done using twosystems to separately sense images.

For example, a group of cells G1, G3, G5, and G7, and a group of cellsG2, G4, G6, and G8 can separately sense images. In this case, thevisible light image and reflected light image have different imageresolutions. In case of the visible light image, since four pixels (twoG pixels and one each R and B pixels) form one color pixel, the matrixshown in FIG. 9 includes 4×4 pixels. When the differences between G1 andG2 and between G3 and G4 are calculated, and are added to each other,i.e., when G1 through G4 form one pixel of the reflected light image,the resolution is halved in the vertical direction, and remains the samein the horizontal direction. When G1 through G8 form one pixel of thereflected light image, the resolution is halved in both the vertical andhorizontal directions (2×2 in FIG. 9).

In the above example, the reflected light image is obtained using “G”cells alone. However, when the three types of cells, i.e., R, G, and Bcells have nearly the same spectral sensitivities to the light sourcewavelength, one pixel of the reflected light image can be formed usingthese four pixels. For example, G1 and R1 cells storenon-emission-timing charges, and B1 and G3 cells store emission-timingcharges. Then, the differences between B1 and G1, and between G3 and R1are added to each other, thus obtaining a reflected light image havingthe same resolution as that of the visible light image. Furthermore, ifthe difference between B1 and G1 or between G3 and R1 is considered asone pixel, the horizontal direction can be doubled with respect to thevisible light image.

Even when “G” cells only can be used to form a reflected light image, areflected light image having the same resolution as that of the visiblelight image can be obtained. If a filter matrix shown in FIG. 10 isused, and if G1 stores a non-emission-timing charge and G2 stores anemission-timing charge, their difference can be output. Since thedifference circuit is normally designed to calculate the differencebetween pixels in an identical line, when the filter matrix shown inFIG. 9 is used, it is difficult to output the difference between G1 andG3, but it is easy to output the difference between G1 and G2 in FIG.10.

Since the above-mentioned control for obtaining a reflected light imageis different from that for obtaining a visible light image, the sensorcontrol must be switched. The control may be explicitly switched by ahardware switch or a software switch (when this apparatus is connectedto a personal computer and is used). Alternatively, a mechanism forturning on/off the switch upon sliding the filter may be provided, andthe current position of the filter may be automatically recognized toswitch the control.

In case of the arrangement shown in FIG. 8, it is easy to separatelyobtain a reflected light image and visible light image but it isdifficult to simultaneously obtain them. However, these images can besimultaneously obtained when a mechanism for moving the optical filterat high speed is added. Also, a method of rotating a circular filterhaving semi-circular filter elements is available.

FIG. 11 shows another example of the arrangement that can simultaneouslysense a reflected light image and visible light image using a singlesensor. Using a light ray splitting means 34 such as a dichroic mirroror the like, which is similar to that used in FIG. 7, light is splitinto object light of the light source and other light components.Shutter means 35 and 36 which can selectively pass or intercept theselight components are provided. The light components that have passedthrough the shutter means are synthesized, and the synthesized light isguided to an optical sensor 38 near-infrared light and reflected lightare synthesized with a light ray splitting means 37.

The shutter means can use liquid crystal panels or the like. Sinceliquid crystal panels or the like can be non-mechanically driven,reliability can be improved as compared to a mechanical structure.

In FIG. 11, object reflected light 39 of the light source isintercepted, and other light components 40 can pass through the shutter.In the other state, the object reflected light reaches the sensor, butvisible light is intercepted by the shutter.

In FIG. 11, two split light beams are synthesized again via thecorresponding shutter means. If an element, the filter characteristicsof which can be dynamically changed, is used, the optical path need notbe split. In an arrangement shown in FIG. 12, an element 41 which can beswitched between two states:

(1) only the light source wavelength is passed; and

(2) only visible light is passed is inserted between the lens andsensor. By only switching the element state, two different types ofimages can be sensed. In FIG. 12, object reflected light 39 of the lightsource is intercepted, and other light components 40 can pass throughthe element. In the other state, the object reflected light reaches thesensor, but visible light is intercepted by the shutter.

If an element which can be switched between two states:

(1) only the light source wavelength is passed; and

(2) all light components are passed and an element which can be switchedbetween two states:

(1) only visible light is passed; and

(2) all light components are passed are available, the same arrangementcan be realized by superposing these elements.

When the color filter of the sensor has a special arrangement, both areflected light image and visible light image can be obtained by asingle sensor. FIG. 10 shows an example of the normal color filtermatrix. However, when a matrix shown in FIG. 13 is used, cells can beclassified into those for a reflected light image, and those for avisible light image. In FIG. 13, cells indicated by “H” have filters forpassing only the wavelength of the light source. R, G, and B cells areused for sensing a visible light image, and H cells are used for sensinga reflected light image.

For example, H1 and H3 cells store non-emission-timing charges, and H2and H4 cells store emission-timing charges. When H1 to H4 cells form onepixel of the reflected light image, the obtained reflected light imagecan have the same resolution as that of the visible light image. In anormal imaging camera, a filter for cutting infrared light components isused. When near infrared light is used as the light source, the lightsource emits light with higher intensity or infrared cut filters areattached onto only cells for the visible light image.

FIG. 14 shows another example of the filter matrix. As compared to FIG.13, the vertical resolution is doubled, and the horizontal resolution isreduced to ⅔. If an image with higher resolution is required, thismatrix is preferably used.

The present invention described above can be improved by softwareprocessing using a versatile processor without using any dedicatedhardware. For example, the processing shown in FIG. 4 can be implementedusing a computer program, and when the computer program is installed inand executed by a computer via a recording medium such as a floppy disk,CD-ROM, or the like, the present invention can be practiced.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

1. An information input method comprising: irradiating an object withlight; picking up, using an image sensor, at an emission time, pickingup at a non-emission time, and picking up a visible light image;generating a timing signal comprised of a pulse signal or a modulationsignal for controlling an intensity of light of a light emitter;generating a control signal for individually controlling light-receivingtimings of light-receiving cells of with image sensor on the basis ofthe timing signal from a timing signal generator; detecting a differencein accumulated electrical charges between a cell of first cells and acorresponding cell of second cells; passing using a filter only lightemitted by said light emitter; intercepting light emitted by said lightemitter; and selecting one of pass filter and cut filter upon passinglight to be sensed.
 2. An information input method, comprising:irradiating an object with light; picking up, using an image sensor, atan emission time, picking up at a non-emission time, and picking up avisible light image; generating a timing signal comprised of a pulsesignal or a modulation signal for controlling an intensity of light of alight emitter; generating a control signal for individually controllinglight-receiving timings of light-receiving cells of with image sensor onthe basis of the timing signal from a timing signal generator; detectinga difference in accumulated electrical charges between a cell of firstcells and a corresponding cell of second cells; passing using a filteronly light emitted by said light emitter; intercepting light emitted bysaid light emitter; and selecting one of pass filter and cut filter insynchronism with the timing signal from said timing signal generator. 3.An information input method, comprising: irradiating an object withlight; picking up, using an image sensor, at an emission time, andpicking up at a non-emission time; generating a timing signal comprisedof a pulse signal or a modulation signal for controlling an intensity oflight of a light emitter; generating a control signal for individuallycontrolling light-receiving timings of light-receiving cells of withimage sensor on the basis of the timing signal from a timing signalgenerator; detecting a difference in accumulated electrical chargesbetween a cell of first cells and a corresponding cell of second cells;picking up a visible light image; splitting light into the objectreflected light and the light other than the object reflected light;selecting whether the light is to be passed or intercepted on opticalpaths of the split light beams; and synthesizing the two light beamssplit by a light splitting section.
 4. An information input method,comprising: irradiating an object with light; picking up, using an imagesensor, at an emission time and picking up at a non-emission time;generating a timing signal comprised of a pulse signal or a modulationsignal for controlling an intensity of light of a light emitter;generating a control signal for individually controlling light-receivingtimings of light-receiving cells of with image sensor on the basis ofthe timing signal from a timing signal generator; detecting a differencein accumulated electrical charges between a cell of first cells and acorresponding cell of second cells; picking up a visible light image;splitting light into the object reflected light and the light other thanthe object reflected light; selecting one of a state wherein only alight source wavelength is passed and a state wherein only visible lightis passed; selecting whether the light is to be passed or intercepted onoptical paths of split light beams; and synthesizing the two light beamssplit by a light splitting section.
 5. An information input method,comprising: irradiating an object with light; picking up, using an imagesensor, at an emission time and picking up at a non-emission time;generating a timing signal comprised of a pulse signal or a modulationsignal for controlling an intensity of light of a light emitter;generating a control signal for individually controlling light-receivingtimings of light-receiving cells of with image sensor on the basis ofthe timing signal from a timing signal generator; detecting a differencein accumulated electrical charges between a cell of first cells and acorresponding cell of second cells; picking up a visible light image;splitting light into the object reflected light and the light other thanthe object reflected light; selecting one of a state wherein only alight source wavelength is passed and a state wherein all lightcomponents are passed; selecting one of a state wherein only visiblelight is passed and a state wherein all light components are passed;selecting whether the light is to be passed or intercepted on opticalpaths of the split light beams; and synthesizing the two light beamssplit by a light splitting section.